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Abstract 


A student’s ranking in the grade point average (GPA) distribution has 
emerged as an admission variable that increases admission rates of both segre- 
gated minorities and high-performance individuals. In 2012, Chile’s centralized 
university admission system introduced a GPA ranking variable relative to the 
previous cohorts’ average GPA. Such a system introduces academic incentives 
to exert effort, but also to inflate GPA. In this paper, we analyze the effects of 
that reform on the GPA distribution and achievement measures. Our results, 
based on difference-in-differences and simulated instruments methods, suggest 
that: (i) GPA increased across the entire distribution, independently of the 
incentive structure; and (ii) GPA increases were unrelated to improvements in 
achievement. We interpret the results as evidence that the introduction of the 


new variable caused GPA inflation rather than increased learning. 
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1 Introduction 


In the United States, a major focus of higher education policy has been to increase 
the enrollment rates of racial minorities, particularly at flagship institutions. To 
achieve this goal, higher education institutions introduced explicit affirmative action 
policies. However, in some cases (e.g., Hopwood v. Texas, 1996; Johnson v. University 
of Georgia, 2001), courts have declared such policies unconstitutional. In response, 
policymakers have changed admission criteria to include observable variables that are 
correlated with academic skills, but independent of race or family income. 

For example, after the ruling of Hopwood v. Texas, Texas introduced for all its 
public universities guaranteed admission to students with GPAs within the top 10% 
of each high school.' Given that Texas high schools have high racial segregation, 
within-school ranking would be a performance indicator uncorrelated with race or 
family income (Cortes, 2010). However, the introduction of such policies can have 
effects beyond admissions: on the one hand, ranking-based admission increases the 
marginal reward of studying and thus might increase learning; on the other hand, it 
can incentivize strategic behavior to manipulate the ranking variable (Cullen et al., 
2013). 

In this paper, we study the effects of a similar nationwide admission policy on high 
school students’ academic performance in Chile. Although Chile is largely racially 
homogeneous (Alesina et al., 2003), socioeconomic status (SES) is an especially strong 
determinant of achievement in a country with high SES segregation across schools 
(Mizala and Torche, 2012). Meanwhile, university admission is based on the Prueba 
de Seleccién Universitaria (PSU) admission exam, which replicates the SES segrega- 
tion pattern and results in highly unequal access to flagship institutions. For example, 


37.7% of the 2011 entering class at Universidad de Chile, the flagship public univer- 


'In Fisher v. University of Texas at Austin (2016), the court upheld affirmative action for 
admission of students below the top 10%. 


sity, came from unsubsidized private schools,? where students from top decile income 
families represent the largest share by far of enrollment (Mizala and Torche, 2012). 
Even higher numbers are observed at Pontificia Universidad Catolica de Chile, the 
flagship private university, where 65.9% of the 2011 entering class came from private 
unsubsidized schools. 

It is this unequal access to flagship institutions, in addition to the financial pres- 
sure that tuition fees put on families, that precipitated the 2011 Chilean students’ 
movement for education reform.® One of the policy responses to the student move- 
ment was the introduction of within-school GPA ranking in the admission criteria of 
the main public and private universities. In particular, the new admission criteria 
increased the importance of high school GPA and introduced a score bonus to re- 
ward students with GPAs exceeding the average GPA of their high school’s last three 
graduating cohorts. 

The potential effects of such a policy can be summarized in three hypotheses. 
First, the new criteria may increase incentives for students to study harder, which 
should result in higher achievement. Second, the new criteria may increase schools’ 
incentives to engage in strategic behavior, such as artificially inflating grades to in- 
crease their students’ university admission rates. Third, the new criteria may induce 
strategic behavior by students in their choice of high school. In this paper, we provide 
empirical evidence about the first two hypotheses. Using a difference-in-differences 
identification strategy, we observe that a cohort exposed for three years to the new 
admission criteria increases its 12th grade GPA by about 0.164 points, or 30.7% of a 
Chilean GPA standard deviation. Later, we use simulated instruments to test if the 
marginal incentives, in the form of a marginal score bonus for a subset of students in 


the GPA distribution, produce different effects. Our findings do not show evidence 


2Source: Consejo Nacional de Educacién - Indices database, available at 
https: //www.cned.cl/indices/ficha-institucional-individual-anos-2007-2016. 
3See With Kiss-Ins and Dances, Young Chileans Push for Reform (New York Times, 2011). 


of differences in GPA increases within a cohort. Instead, they show that the GPA 
increase took place across the entire cohort. 

To test whether the GPA increase may be due to increased learning, we investi- 
gate whether GPA increases predict changes in standardized test scores or university 
admission exam taking rates or performance. None of these achievement indicators 
show any significant correlation with the GPA increase, suggesting that the GPA in- 
crease is due to grade inflation rather than increased learning. In consequence, these 
results suggest that schools have adjusted grading strategically in response to the new 
university admission criteria. 

The paper is organized as follows. Section 2 describes the Chilean university 
admission system. Section 3 reviews the literature related to ranking-based admission. 
Section 4 provides a simple conceptual framework to understand students’ decisions 
under the new admission criteria. Section 5 describes the data and methodologies used 
in the paper. Sections 6 and 7 provide the estimated effects of the new admission 


criteria on GPA and achievement, respectively. Section 8 concludes. 


2 University admission in Chile 


2.1 Background 


To apply to universities in Chile, students submit an ordered list of their selected 
program-university combinations to a centralized and transparent system, which fills 
the available spaces with the applicants with the highest application scores.* The 
main variable in the application score is the Prueba de Seleccién Universitaria (PSU) 
university admission exam score, which is similar to the SAT exam of the United 


States. PSU scores range from 150 to 850 points, with an average of 500 points and a 


4For further details about the application system, see Hastings et al. (2013). 


standard deviation of 110 points. Every student applying for university admission is 
required to take the two PSU core tests (Math; and Language and Communications) 
and at least one of two PSU subject tests (Science; and History, Geography, and 
Social Sciences). Before 2012, universities assigned every applicant an application 
score based solely on PSU scores and high school GPA, usually weighted at an 80:20 
ratio, respectively. 

However, one problem of PSU-based admission is the high correlation between 
PSU scores and family income. To illustrate this, Figure l(a) shows the geographical 
distribution of 2010 PSU results, defined as the probability of scoring above 450 
points,’ across municipalities in the city of Santiago. As we can see, there is much 
geographical segregation in PSU performance. In most eastern municipalities, more 
than 64% of students score above 450 points. Meanwhile, less than 35% of students in 
many western and central-southern municipalities score above 450 points. As shown in 
Figure 1(b), municipal segregation patterns in PSU performance align with municipal 
poverty rates. The higher scoring eastern municipalities have lower poverty rates 
(at most 8%), while the lower scoring western and central-southern municipalities 
have higher poverty rates (at least 16%). In sum, geographic segregation in PSU 
performance mirrors geographic segregation in income, showing that PSU scores are 
strongly correlated with family income. 

In response to Chilean students’ demand to break the link between family income 
and university admission, the Consejo de Rectores de las Universidades Chilenas 
(CRUCH) announced in 2012 a variant of GPA ranking for introduction to the ad- 
mission criteria beginning in 2013. As in Texas, such a change should increase the 
admission rates of disadvantaged groups, without explicitly using SES as an admis- 


sion criterion (Cortes, 2010). In addition, there is evidence that high school GPA 


>A minimum score of 450 is required to apply to CRUCH institutions (the most traditional 
universities) and to qualify for government student aid. For more details, see Solis (2017). 


Figure 1: PSU performance and poverty rates across municipalities in Santiago, 
Chile, 2010. 
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(a) Probabilities of PSU score>450 (b) Poverty rates 


Source: Ministerio de Educacién and Ministerio de Desarrollo Social, Chile. 


ranking is a good predictor of university performance (Contreras et al., 2009, 2017). 
As a result, the introduction of high school GPA ranking to the admission criteria 


was regarded as a good policy by both government and students representatives. 


2.2 The Chilean GPA ranking score 


As mentioned above, the main state and private universities, which comprise CRUCH, 
introduced a high school GPA ranking score to the 2013 admission criteria. While 
universities’ admission criteria have always included GPA in application scores, with 
weights ranging between 20% and 40% across programs, the new 2013 admission 
criteria set a 10% weight to a new GPA ranking score (Demre, 2012). Given the 
competitive structure of the Chilean university admission system, such a percentage 
is enough to make the GPA ranking score important in university admission. 


There are important differences to note between the US and Chilean definitions 


of GPA ranking. First, as policymakers did not want to induce competition between 
classmates (Gil et al., 2013), the Chilean GPA ranking score is a function of the 
average GPA of a high school’s three previous graduating cohorts (historical average) 
rather than the current graduating cohort.° Second, the function translating Chilean 
GPA to application scores is not a step function, as in the case of Texas. Instead, 


GPA in Chile translates to a ranking score r(x) that is the sum of: 


1. A linear function z(#) that converts GPA to a score ranging from 150 points 
for a 4.0 GPA (the minimum passing grade) to a maximum of 850 points for a 


7.0 GPA (a perfect score);’ and 
2. A score bonus b(x), which is defined as: 


(a) b(x) = 0, if the student’s GPA is below the historical average (Zprevious); 


(b) b(x2) > 0, if the student’s GPA is above the historical average and below 


the average GPA of the three previous valedictorians (Max(2previous))} OF 


(c) b(x) = 850 — z(x), if the student’s GPA is above the average GPA of the 
three previous valedictorians (Max(2previous)), Which results in a perfect 


ranking score (r(x) = 850). 


To illustrate the ranking score scheme, Figure 2 shows the relationship between 
GPA and the ranking score. The score bonus is the distance between the solid z(x) + 
b(a) ranking score line and the dashed z(x) line. Consequently, the closer the historical 
average and the maximum GPA are, the greater the marginal score bonus is. As we 
can see, such a scheme encourages schools to engage in strategic behavior in the 


form of GPA inflation, but also provides an extra incentive to students with GPAs 


SLater, this was changed to the three previous cohorts’ graduates from all high schools where 
the student was enrolled (Gonzalez and Johnson, 2018). 

"For a conversion table showing how GPA is converted to an application score, see Tabla 
Transformacién NEM. 


above the historical average to exert effort, which allows us to analyze the effects of 
variations in the marginal incentives. 
Figure 2: Ranking score scheme. 
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3 Literature review 


The literature examining ranking-based admission programs focuses mainly on Texas’ 
Top 10% GPA university admission policy. Despite its popularity, Cortes (2010) shows 
that the top 10% admission policy was unable to replicate the benefits of affirmative 
action policies to underrepresented minorities, particularly among minority individ- 
uals below the top 10% that would have been admitted under affirmative action 
policies. Meanwhile, Cullen et al. (2013) provide evidence of strategic choice of high 
schools to enter the top 10% GPA group. Similarly, Cortes and Friedson (2014) pro- 
vide evidence of real estate price increases near low-performing schools being driven 
by demand for increased probability of university admission. 

There is less literature about the effects of GPA ranking-based admission on 


achievement. If GPA captures student effort better than admission exam scores, then 


GPA-based criteria will provide greater incentive to exert effort than exam-based 
criteria. Cortes and Zhang (2011) provide evidence that Texas’ top 10% admission 
policy had incentive effects on students’ school performance; that is, increases in effort 
among students in low-performing schools exceeded increases in effort among students 
who would have been admitted regardless of the policy. The authors also found that 
students from low-performing schools took less challenging upper-level courses.® 

Related literature examines the effects of financial incentives on achievement. Kre- 
mer et al. (2009) find that financial incentives to score in the top 15% on academic 
exams had positive effects on achievement as well as positive externalities to non- 
incentivized students. Meanwhile, Bettinger (2012) finds significant positive effects 
from financial incentives on math test scores, but not on other subjects. In New York, 
Chicago, and Dallas, Fryer Jr (2011) finds little effect on state test scores from finan- 
cial incentives to read books and improve test scores and classroom grades, despite 
multiple experimental interventions. Finally, Hirshleifer (2015) finds that financial 
incentives to increase inputs generate greater achievement gains than incentives to 
increase outputs, which suggests that students’ myopic behavior makes GPA-based 
admission more incentivizing to effort than test score-based admission. 

With regard to Chile, Contreras et al. (2017) evaluate a 2007 CRUCH policy that 
gave special admission to students with GPAs in the top 5% of each high school who 
scored within 5% of the application score cutoff. Using a regression discontinuity 
design, they find that these special admission students performed well academically, 
in terms of retention, due to non-cognitive skills. In consequence, they suggest that 
the GPA ranking-based criterion would achieve equity goals without efficiency costs. 

In analyzing the effects of the introduction of the ranking score, Gonzalez and 


Johnson (2018) present descriptive evidence of high school GPA increases relative 


8Song (2017) provides evidence of increases in students’ school performance due to a policy in 
China guaranteeing high school admission to the top 10% GPA students in each middle school. 


to the GPA at the last middle school grade level. They also find little evidence 
of strategic school choice, observing that each year only 0.7% of students switched 
schools to increase their application scores. Thus, the authors conclude that strategic 
behavior by students or their parents has not been prevalent thus far. 

In sum, the literature provides some support to the idea that incentives might 
result in higher achievement and strategic school choice. However, little has been 


written about strategic behavior by schools. 


4 Conceptual framework 


In order to formalize the effects of the new admission criteria, this section introduces 
a simple model of a student deciding for what GPA zx to study. First, We assume 
that the student has a skill 6, such that an effective study effort of 1/0 delivers one 
GPA point. In addition, we assume that: (1) the probability of admission P() is 
only a function of the GPA 2; (2) the expected utility given admission is vz; and (3) 
the expected utility given non-admission is vz. Thus, the student’s expected utility 
is: 


u(z,0) = P(#)-vy + (1— P(x))- vu, -— ¢ (5) (1) 


where ¢(-) is the disutility of effective study effort. For the problem to have a simple 
solution, we need ¢(-) to be convex and P(x) to be increasing (higher GPA improves 
admission probabilities) and concave. Also, we assume that vy > vz, so that enroll- 
ment is always preferred given admission. 

As we know, the optimal decision of the student is characterized by the first order 


condition. Therefore, the student should equalize the increase in the expected utility 


10 


due to a marginal increase in GPA, with the marginal disutility of study effort: 
/ 1 / ag 
P'(2)- (vy — vz) = 5-4 (Z) (2) 


This simple result summarizes the idea that students should change their behavior 
with the marginal incentives, which in this case is P’(x)-(vy—vz). As the introduction 
of the ranking score increases P’(x), we expect that the new admission criteria induce 
more student effort and, therefore, higher achievement. 

To analyze the consequences of the unusual Chilean policy assigning application 
score bonuses to students with GPAs above the historical average, Figure 3 presents 
the optimal choice for a student whose GPA exceeds the historical average, with 
and without the score bonus. Similar to a graph of income before and after tax, 
Figure 3 shows that individuals have a preference for the ranking score (vertical 
axis), but to increase it they must exert effort represented by the GPA (horizontal 
axis). Meanwhile, a government schedule annually assigns ranking scores to GPAs 
within each high school. Without the score bonus, the schedule would be z(x), but 
with the score bonus, the schedule is z(2)+6(x). The student should choose the point 
where the indifference curve, which increases utility upward to the left, is tangent to 
the government schedule. That point is in the indifference curve U,, for a schedule 
without the score bonus, and in the indifference curve wu, for the real schedule with 
the score bonus. If the substitution effect dominates the income effect (which is equal 
to increases in admission probabilities diminishing effort), we can expect that the 
presence of the score bonus increases GPA by Az. 

If we assume that P’(x) = a is in the neighborhood of a point xz and that the 
values of a vary in the data, then we can estimate the relationship between a and 
x. In other words, we can estimate the elasticity of GPA with respect to marginal 


academic incentives. 


yb 


Figure 3: Student decision under different GPA-ranking score schedules. 
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5 Data and methodology 


The first goal of the empirical analysis is to estimate the effect that the introduction 
of the ranking score had on high school GPAs. However, since it is possible that GPAs 
change due to inflation rather than increased student effort or learning, a second goal 
is to look at the relationship between GPA increases and changes in other achievement 
measures. To achieve that goal, we relate at an aggregate (municipality) level the 
effects of the policy with the changes, between the years 2010 and 2014, on the 10th 
grade universal standardized test (SIMCE) and PSU admission exam results. For 
that purpose, this section will describe the data, provide graphical evidence of the 


effects, and discuss the methodological approach. 


5.1 School Data 


To measure students’ school performance, we use publicly available administrative 


data on students that graduated high school in Chile between 2011 and 2014, including 
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all year-end individual GPAs after 7th grade for the period 2007 to 2014, linked to 
basic individual-level demographic information. In the analysis, we use the 8th grade 
(last middle school grade) GPA to construct a pre-ranking measure of individual 
performance. Then, we use the change in GPA in each grade between 8th and 12th 
as a dependent variable. Finally, the school identifier can be used to add school- 
level information, such as the ranking parameters, the type of school (public, private 
subsidized, or private non-subsidized), geographic location (urban or rural) and a 
general measure of socioeconomic status (SES). 

Table 1 shows basic statistics on the main variables used in the empirical analysis. 
We have around 730,000 students at each grade level, associated with 2.9 million 
annual observations.? The average GPA (on a scale of 1 to 7) is slightly above 5.5, 
with a standard deviation around 0.535. Public and private subsidized schools are 
similarly represented in the database (43% and 51% of the total, respectively). Of 
the students, 34.5% are tagged as low SES, almost half are male, and 3.6% come from 
rural schools. 

Given our interest in measuring the changes in the GPA used to apply for univer- 
sity admission, a number of filters were applied to the original dataset. In addition 
to requiring a balanced panel, we dropped any individuals with contradictory data 
and any years in which a student did not pass (the GPAs of which are excluded from 
university application GPAs in Chile).'° 

To diminish potential bias from individuals that strategically choose high schools, 
we only use data from individuals that had already entered high school at the time of 
the policy announcement. In addition, Gonzdlez and Johnson (2018) estimate that 


the use of the ranking score for university admission induced a during-high school 


°The working database represents a balanced panel, with students observed in all 4 years of 
high school. 

10More details about the data cleaning are presented in Appendix Table A.1. The graphical 
analysis with the full sample is in Appendix Figure A.3. 
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Table 1: Summary statistics. 


Variable Mean Std. Dev. N 
Ranking score in place 0.382 0.486 2,906,544 
9th grade GPA 5.492 0.534 726,636 
10th grade GPA 5.537 0.535 726,636 
11th grade GPA 5.557 0.535 726,636 
12th grade GPA 5.654 0.548 726,636 
Dummy year 2008 0.063 0.243 2,906,544 
Dummy year 2009 0.126 0.3382 2,906,544 
Dummy year 2010 0.188 0.391 2,906,544 
Dummy year 2011 0.241 0.427 2,906,544 
Dummy year 2012 0.19 0.392 2,906,544 
Dummy year 2013 0.128 0.334 2,906,544 
Dummy year 2014 0.064 0.245 2,906,544 
Public school 0.434 0.496 2,906,544 
Private school, subsidized 0.512 0.5 2,906,544 
Private school, non-subsidized 0.054 0.227 2,906,544 
Low SES student 0.345 0.475 2,906,544 
Male student 0.478 0.5 2,906,544 
Rural school 0.036 0.187 2,906,544 


migration of only 0.7%. Given the low prevalence of during-high school ranking- 
induced migration and the difficulty in identifying ranking-induced migration versus 
migration for other reasons, we do not drop individuals that have migrated schools 
after entering high school. 

The theoretical model suggests that the introduction of the ranking score could 
affect the performance of all students, but especially of those who have a pre-ranking 
GPA between the average and maximum GPA of the three previous graduating co- 
horts. To graphically evaluate its effect over the GPA distribution, we categorize 
students by their 8th grade GPA quintile within the same school and cohort. Figure 
4 shows the average change in yearly GPA (with respect to the previous grade level) 
as a function of the year and 8th grade GPA quintile. We include 4 years of analysis 
per grade level, which corresponds to students that should graduate between 2011 and 


2014. Because the ranking score was announced in 2012, only observations after that 
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year could be affected by the policy; those include the GPAs of 10th grade students 


in 2012, 11th grade students in 2012-2013, and 12th grade students in 2012-2014. 


Figure 4: Change in yearly GPA by 8th grade GPA quintile, years 2008-2014. 
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A number of patterns appear from this graphical analysis. First, Figure 4(a) 
shows that, in general, GPA decreases when entering high school, especially among 
students in the upper part of the GPA distribution. More importantly, the change in 
yearly GPA between 8th and 9th grade appears essentially the same for the 4 cohorts 
under analysis in 2011 before the ranking score was announced. A second pattern 
is that after 9th grade, GPA increases over time for almost all quintiles in all years. 
However, the most salient observation from Figure 4 is that the GPA increases are 


higher after the announcement of the ranking score in 2012. In other words, the 
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graphical analysis suggests that the introduction of the ranking score had positive 
effects on GPA. In fact, the observed effect seems to be similar over the entire GPA 
distribution. Even if the quintiles are an imperfect measure of the student’s position 
relative to the mean and maximum GPA at the end of high school, we would have 
expected greater effects in the upper half of the distribution. We will further pursue 


this finding in the empirical analysis. 


5.2 Empirical methodology 


Typically, we could use a simple difference-in-differences estimation model with cohort 


and grade level fixed effects, such as: 
AGPA yy, = 09 + 6+: Petmtuyt+ ex (3) 


where 7 represents the individual, / represents the grade level, t represents the cohort, 
and Py is a “ranking score in place” dummy variable. 
However, formalizing the graphical analysis in Figure 4 requires the inclusion of 


dummies Q} — Q? for each quintile of 8th grade academic performance. 


5 
AGPAin = 0% + ee bg Pu Q) +5 (ma QY) tu + eu (4) 
q=1 q=1 
This model includes a joint quintile/grade level fixed effect 7, which captures the 
average change in GPA before the ranking score, at grade 1, for quintile g of 8th grade 
academic performance. Meanwhile, the interaction term P, - Q? indicates whether 
the ranking score is in place for quintile g, and accordingly, coefficient 6, estimates 
the effect of the ranking score introduction for each quintile gq. 
As suggested in Section 4, greater marginal incentives may result in greater GPA 


increases. To test that hypothesis, we estimate models that include the marginal 
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incentive faced by each student, which we measure as the marginal score bonus that a 
student would receive if she increased her GPA slightly. The model includes a single 
post-reform dummy (Py), a set of quintile dummies (Q?), and the extra marginal 
incentive that the student faces (B,,). We also control for individual level covariates 
with dummies (X;j;), such as for low SES, gender, and the enrolled school type (public, 


private subsidized, or private non-subsidized): 


5 
AGPAin = 09+ 6+ Py t+ B+ Bat >> (ma: Qf) +4 + O'Xa + Eu. (5) 
q=l 

Given that the introduction of the ranking score affects all students, the coefficient 6 
estimates the average effect of the policy among students that do not face a marginal 
score bonus. Meanwhile, the coefficient 3 estimates whether changes in the marginal 
score bonus (the slope of b(x) in Figure 3) induce changes in GPA. If marginal incen- 

tives are indeed effective, as shown in Figure 3, then ( will be positive. 
Nevertheless, our OLS estimates of Equation 5 are potentially biased, since stu- 
dents might endogenously choose their marginal incentives by changing their GPAs! 
or by switching high schools. Therefore, we also estimate two-stage least squares 
(2SLS) models using as an instrument a simulated marginal score bonus based on the 
ranking score formula, the 8th grade GPA (which is not part of the ranking score), 
and the average and maximum GPAs of the student’s 9th grade high school (chosen 
before ranking scores were announced). The advantage of such a simulated instru- 
ment is that it is based on variables that should not be affected by the posterior 
introduction of the ranking score. With those variables at hand, we calculate the 


simulated marginal score bonus in the following steps: 


1. Simulate the high school GPA: Using the 8th grade GPAs and assuming 


'This is similar to the problem of endogeneity of marginal income tax rates after labor supply 
adjustments. 


it 


that GPAs evolve like they did for the 2011 cohort (before rankings were an- 
nounced), we simulate the GPAs of the 2012 — 2014 cohorts in the 9th, 10th, 


11th, and 12th grade levels. 


2. Simulate the ranking score: We use the simulated GPAs and the ranking 
score formula based on the parameters of the student’s 9th grade high school 


to calculate simulated ranking scores. 


3. Simulate the marginal score bonus: Finally, we calculate the simulated 
marginal score bonus as the difference between the simulated marginal incen- 


tives with and without the score bonus. 


In sum, we build a simulated instrument that represents the extra incentives a 
student in the bonus area faces before she makes adjustments.'* Therefore, the iden- 
tification comes from the student’s initial GPA and the particular ranking parameters 
(her school’s average and maximum historical GPA) that she faces. Finally, we must 
note that the main assumption of our 2SLS estimator is that the simulated instrument 
is orthogonal to the error term, given the 8th grade GPA.'? Therefore, we include the 


quintile dummies Q? to control for the student’s normal level of effort or ability. 


5.3 Standardized tests 


Figure 4 shows that the introduction of the ranking score affected GPA, and its 
effects appear homogeneous across the GPA distribution. Such a result is consistent 
with a general increase in student effort after the introduction of the ranking score. 
However, the result is also consistent with schools inflating GPAs in order to improve 


their students’ prospects, which may also benefit schools in the competitive Chilean 


!2This is similar to the income tax rate that an individual faces before adjusting labor income 
to a tax rate change. 

13T other words, we assume that the changes in marginal incentives are exogenous, given 
previous academic performance. 


18 


school market. To disentangle these alternative hypotheses, we analyze the correlation 
between the GPA increases and the changes in other achievement statistics, since 
increased student effort should increase achievement, while GPA inflation should not. 

As we are not able, with public data, to match individual-level GPA to individual- 
level achievement measures, we use municipality-level achievement measures and es- 
timate the effects of the ranking score on GPA for each municipality. Specifically, we 
use achievement results from two standardized tests in the years 2010 (pre-ranking) 
and 2014 (post-ranking): the 10th grade SIMCE national standardized test (scores) 
and the PSU university admission exam (take-up rates and scores).'4 

In practice, we use Equation (3) to estimate the average GPA increase in each 
municipality. Since we want a certain precision in our estimates, we do not include 
quintile dummies and restrict our analysis to municipalities with at least 250 GPA 
observations in the sample.!° As a final step, we use the municipality-level estimates 
to show both graphical and OLS evidence of the correlation between the ranking score 


effects on GPA and the variations in SIMCE/PSU achievement measures. 


6 Results 


In this section, we present estimation results for the two types of analyses described in 
the previous section: a difference-in-differences estimation of the effect of the ranking 
score introduction on GPA; and the OLS and 2SLS estimations of the effect of the 


marginal score bonus incentive on GPA. 


14The PSU admission exam is taken after graduation and administered on a national level. 
‘Other binding criteria provide similar results. 
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6.1 Difference-in-differences results 


Table 2 presents the OLS results of the difference-in-differences model. In columns 
(1) and (2), the first row coefficients show the estimated average effect of the ranking 
score on GPA. Our results suggest that the policy increased GPA, from one year to 
the next, by approximately 0.054 points.!° Assuming a GPA standard deviation of 
0.535 (Table 1), the estimated effect is equivalent to a yearly increase of 10.1% of a 
GPA standard deviation. Table 2 also shows that the yearly change in GPA is, on 
average, smaller for low SES students, male students, and students enrolled in private 
schools (especially in non-subsidized institutions). 

Also in Table 2, columns (3) and (4) show the estimation results with five in- 
teraction dummies of the ranking score policy with the within-school quintile of 8th 
grade GPA, which allows us to examine the heterogeneity in effects. In both spec- 
ifications, the introduction of the ranking score seems to have increased GPA for 
students throughout the entire distribution, although with slightly greater effects in 
the intermediate quintiles. In particular, the last row shows that the hypothesis of 
homogeneous effects is rejected at a 1% significance level. 

Table 3 shows the estimation results of the following re-parameterized version of 
the difference-in-differences model, which captures the total effect between the 10th 


and 12th grade levels in a single coefficient (;: 
5 
AGPAjny = a0+ B1- Py 1]? +62: Pie (079-17?) +63 Pie (IP) +> (ma QD +u+e1 (6) 
q=1 


where the dummy variable 1 indicates whether the grade level | of the observation 
is equal to g. Most importantly, the coefficient (, corresponds to the sum of effects 
in the 10th, 11th, and 12th grade levels, whereas coefficients G2 and (3 estimate the 


one year effects in the 10th and 11th grade levels, respectively. 


16Results of similar size, available in Appendix Table A.2, were found with simple grade level 
dummies, instead of quintile-level interaction dummies. 
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Table 2: Effect of ranking score introduction on yearly change in GPA, cohorts 


2011-2014. 
@ ©) 8) ® 
AGPA AGPA AGPA AGPA 
VARIABLES per grade per grade’ per grade per grade 
Ranking score in place O.0bs8°** .0587""* 
(0.00426) — (0.00426) 
Ranking effect on quintile 1 O.0ali*** 05144" 
(0.00410) (0.00410) 
Ranking effect on quintile 2 0.0562*** 0.0565*** 
(0.00407) — (0.00406) 
Ranking effect on quintile 3 0.0565"** 0.0564*** 
(0.00428) — (0.00429) 
Ranking effect on quintile 4 0.056077". 0.0557*** 
(0.00461) (0.00461) 
Ranking effect on quintile 5 0.0495***  0.0488*** 
(0.00542) (0.00542) 
Low SES student -0.0286*** -0.0286*** 
(0.00133) (0.00133) 
Male student -0.0181*** -0.0181*** 
(0.00154) (0.00154) 
Private school, subsidized -0.00682** -0.00682** 
(0.00315) (0.00315) 
Private school, non-subsidized -0.0357*** -0.0357*** 
(0.00797) (0.00798) 
Constant -0.373***  -0.361*** — -0.373*** — -0.361*** 
(0.00789) (0.00852) (0.00789) (0.00852) 
Observations 2,906,544 2,906,544 2,906,544 2,906,544 
R-squared 0.161 0.163 0.161 0.163 
Quintile-level dummies Yes Yes Yes Yes 
Cohort dummies Yes Yes Yes Yes 
p-value heterogeneous effects 0.000478 0.000453 


Clustered standard errors at the school level in parentheses. 


*** N<0.1, ** p<0.05, *** p<0.01. 


Finally, Table 3 shows that the estimated 3-year effect is 0.164 GPA points, equiv- 


points, equivalent to 40% of a GPA standard deviation. 


Pal 


alent to 30% of a GPA standard deviation. This result is robust to the introduction 
of covariates, as shown in column (2). Since, in the long term, the ranking scores will 


include all 4 years of high school, we can extrapolate a 4-year effect of 0.219 GPA 


Table 3: Effect of ranking score introduction on yearly change in GPA, re- 


parametrized model, cohorts 2011-2012. 


@ ) 
AGPA AGPA 
VARIABLES per grade _ per grade 
Ranking score in place through 3 grade levels 0.165*** = 0.164*** 
(0.0129) (0.0129) 
Treatment 10th - treatment 12th 0.0477***  0.0476*** 
(0.00516) — (0.00516) 
Treatment 11th - treatment 12th 0.0479***  0.0478*** 
(0.00467) — (0.00467) 
Low SES student -0.0286*** 
(0.00133) 
Male student -0.0181*** 
(0.00155) 
Private school, subsidized -0.00680** 
(0.00315) 
Private school, non-subsidized -0.0357*** 
(0.00797) 
Constant -.371"** -0.359*** 
(0.00789) — (0.00852) 
Observations 2,906,544 2,906,544 
R-squared 0.161 0.164 
Quintile-grade dummies Yes Yes 
Cohort FE Yes Yes 


Clustered standard errors at the school level in parentheses. 
EE <0.1, ** p<0.05, *** p<0.01. 


6.2 Simulated instrument results 


In this subsection, we present the estimation results of the effect on GPA of the 
marginal score bonus incentive, which is the difference between the slopes of the two 
curves depicted in Figure 2: the slope of z(x)+b(x) minus the slope of z(a). Equivalent 
to column (2) of Table 2, the first column of Table 4 presents the simple OLS result, 
which now includes in the first row the marginal score bonus coefficient. Once again, 
we include in the estimation quintile-grade dummies and cohort fixed effects. Given 
that the marginal score bonus variable might be endogenous, the second column of 


Table 4 presents the first stage of the 2S5LS estimation, where the dependent variable 


22 


is the observed marginal bonus and the omitted instrument is the simulated marginal 


score bonus. Finally, the third column presents the second stage results. 


Table 4: OLS and 2SLS effect of the ranking score bonus. 


@ ®) 8) 
AGPA Marginal AGPA 
VARIABLES OLS bonus 2SLS 
Marginal score bonus -0.0389*** -0.00215 
(0.00101) (0.00296) 
Simulated marginal bonus O.24500" 
(0.00539) 
Ranking score in place 0.0689*** 0.276*** 0.0546*** 
(0.00426) (0.00466) — (0.00430) 
Low SES student -0.0276***  0.0139***  -0.0285*** 
(0.00132) (0.00183) (0.00132) 
Male student -0.0178***  0.00432*** -0.0181*** 
(0.00154) (0.00142) — (0.00154) 
Private school, subsidized -0.00707**  -0.00587** — -0.00683** 
(0.00316) (0.00280) (0.00315) 
Private school, non-subsidized -0.0363*** = -0.0185*** — -0.0357*** 
(0.00799) (0.00640) (0.00797) 
Constant 0.0332***  -0.00733*** 0.0336*** 
(0.00457) (0.00178) (0.00456) 
Observations 2,906,544 2,906,544 2,906,544 
R-squared 0.166 0.217 0.164 
Quintile-grade dummies Yes Yes Yes 
Cohort FE Yes Yes Yes 


Clustered standard errors at the school level in parentheses. 
**K D<0.1, ** p<0.05, *** p<0.01. 


The OLS estimation suggests that the introduction of the ranking score had a pos- 
itive effect on GPA, while the marginal score bonus had a negative effect. Specifically, 
the third row coefficient in column (1) indicates that the introduction of the ranking 
score is correlated with a yearly GPA increase of 0.0689 points for students outside 
the marginal score bonus area (low performance students). Meanwhile, the marginal 
score bonus coefficient suggests that these incentives had negative effects on GPA. 


Such a result is counterintuitive and may be due to biases in the OLS estimation. 
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In column (2) of Table 4, the first stage results show that the simulated instrument 
(second row) is strongly related to the endogenous variable and significant at 1%. In 
fact, the Cragg-Donald and Kleibergen-Paap weak identification test statistics reject 
the underidentification hypothesis. As expected, the coefficient of the simulated score 
bonus suggests that our instrument has a positive relationship with the real score 
bonus. 

In opposition to the OLS results, the 2SLS results (in column 3) are more in line 
with the expected effect of the reform.'’ We estimate that the introduction of the 
ranking score increased GPA by 0.0546 points per year, which is significant at 1% 
and equivalent to 10.2% of a GPA standard deviation. Meanwhile, the marginal score 
bonus coefficient is close to zero and not significant at 10%, despite the large sample 
size. The coefficient suggests a GPA decrease of 0.002 points per unit of bonus, which 
is equivalent to 0.48% of a GPA standard deviation, and thus also insignificant in 
practice.'* In conclusion, the main effect comes from the existence of the ranking 


score, rather than the marginal score bonus incentives for a subset of students. 


7 Relationship with other achievement measures 


In the previous section, we showed robust evidence of a generalized increase in GPAs 
due to the introduction of the ranking score, which can be explained by a generalized 
increase in student effort or grade inflation. To distinguish between those hypotheses, 
we study the correlation between the increases in GPAs and the changes in achieve- 


ment measures at the municipality level. 


17 As an alternative way to control for previous performance, Appendix Table A.3 shows similar 
results with linear controls. 

18The average marginal score bonus is 1.184 in the range where it is positive, which suggests an 
effect of -0.002 for the average beneficiary of the bonus. 
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7.1 Effect on standardized test scores 


First, we study the correlation between the estimated effects on GPA and the 10th 
grade national standardized test (SIMCE) scores. Figures 5(a) and 5(b) show scatter 
plots of the changes in SIMCE (Math and Language) scores and the estimated effect 
of the ranking score on GPA. The plots show no pattern in the data and simple linear 
fits show no correlation between variables. 


Figure 5: Graphical evidence: ATest scores 2014-2010 and effect of ranking score 
introduction, municipality-level data. 
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The formal OLS analysis is presented in Table 5, where columns (1) and (2) show 
the estimation results for Language, while columns (3) and (4) show the results for 
Math. Notice also that specifications (2) and (4) include municipality-level controls 
for the student-teacher ratio (education quality measure) and 2010 poverty rate (SES 
measure). In all columns, the correlation between the estimated effect and the change 
in achievement is positive, but small and non-significant.!? Therefore, the results 
indicate that the increases in GPA are not related to increases in achievement; in 
consequence, they fit better the GPA inflation hypothesis than the increased student 


effort hypothesis. 


19 Assuming that the average effect of ranking scores on GPA is 0.05 points, the effect on 
achievement would be 0.32 SIMCE points in Language and 0.16 SIMCE points in Math. The 
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Table 5: OLS evidence: ALanguage and AMath test scores 2014-2010 and effect of 
ranking score introduction, municipality-level data. 


Me) 8) @ 
ALang ALang AMath AMath 
VARIABLES 2014-10 2014-10 2014-10 2014-10 
Estimated effect in municipality 5.167 6.302 1.654 3.144 
(12.50) (12.07) (18.85) (18.03) 
Students per teacher, 2010 1.123*** 2.180*** 
(0.383) (0.509) 
Poverty rate, 2010 -0.456** -0.793*** 
(0.177) (0.256) 
Constant -2.920* -15.99** 6.816***  -20.05* 
(1.511) (7.908) (2.275) ~—(11.23) 
Observations 311 311 312 312 
R-squared 0.001 0.056 0.000 0.087 


Robust standard errors in parentheses 


eee OO. ** peo, * peo 


7.2 Effect on admission exam results 


Next, we study the correlation between the estimated effects on GPA and the Chilean 


university admission exam (PSU) results. The PSU score is the average of the Math 


and Language component scores on a 150-850 point scale. Given that the PSU is a 


voluntary exam, we look at two measures of achievement change: the variation in 


PSU exam taking rate and the variation in the percentage of students scoring above 


450 points.?° 


A graphical analysis is presented in Figure 6. Figure 6(a) shows little correla- 


tion between the estimated ranking score effect on GPA and the PSU exam taking 


rate, while Figure 6(b) shows similar results for the percentage of students scoring 


above 450 points on the PSU. Therefore, the graphical evidence suggests that GPAs 


increased due to GPA inflation rather than an increase in student effort. 


SIMCE test is designed to have a standard deviation around 50 points. 
20 As explained in subsection 2.1, a minimum PSU score of 450 points is required to qualify for 


student financial aid. 
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Figure 6: Graphical evidence: APSU admission exam taking rate and performance 
(2014-10), and effect of ranking score introduction, municipality-level data. 
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Finally, Table 6 shows the regression results for the PSU taking rate (columns 
1-2) and the percentage of students that scored above 450 points (columns 3-4). 
Once again, the analysis suggests that the effect of the ranking score on GPA is not 
correlated with observable increases in alternative achievement measures, since all the 
relevant coefficients are small and not statistically significant.2! Hence, we find no 
evidence of an improvement in the PSU admission exam results, which again suggests 
that the GPA increase was due to GPA inflation, rather than an increase in student 


effort. 


8 Conclusions 


In 2012, Chile announced that it would introduce to its university admission criteria 
a GPA ranking score variable whose characteristics include a score bonus to students 
with GPAs exceeding those of the same school’s previous cohorts. As the new ranking 
score variable increases the importance of the student’s grade point average (GPA), 


students are incentivized to exert more effort. However, the variable also incentivizes 


21 Assuming an average effect of 0.05 GPA points, the average effects are 0.29% in the PSU 
taking rate and -0.16% in the percentage of students scoring above 450 points. 
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Table 6: OLS evidence: APSU admission exam taking rate and performance (2014- 
2010) and effect of ranking score introduction, municipality-level data. 


(1) (2) (3) (4) 
A% stud. A% stud. A% stud. A% stud. 


VARIABLES PSU PSU PSU>450 PSU>450 
Estimated effect in municipality 0.0642 0.0578 -0.0297 -0.0325 
(0.0750) (0.0743) (0.0732) (0.0733) 
Students per teacher, 2010 -0.000339 0.000450 
(0.00192) (0.00149) 
Poverty rate, 2010 0.00196* 0.000676 
(0.00104) (0.000698) 
Constant 0.0570*** 0.0298  0.0493*** 0.0295 


(0.00765) (0.0444) (0.00655) — (0.0352) 


Observations 320 320 320 320 
R-squared 0.003 0.020 0.001 0.005 
Robust standard errors in parentheses 
kK aK * 
p<0.01, p<0.05, * p<0.1 


schools to inflate GPAs to improve their students’ prospects for higher education. 
This paper contributes to the literature by identifying the effects on GPA of the 
introduction of a GPA ranking score variable in university admission criteria, as well 
as by disentangling the sources of those effects. 

Specifically, we study the evolution of GPA before and after the introduction of 
the ranking score and analyze whether the changes in GPA correlate with changes in 
standardized achievement measures at the municipality level. Our main findings are 
three: (i) the introduction of the ranking score produced a yearly increase in GPA 
equivalent to 10.1% of a GPA standard deviation; (ii) the increase in GPA takes place 
throughout the entire GPA distribution, independently of the different incentives 
faced by individuals; and (iii) the GPA increase, when aggregated at the municipality 
level, is not correlated with improvements in other achievement measures. We believe 
all of these findings indicate that some schools opted to artificially inflate student 


GPAs, with no visible increases in student effort or learning. 
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While our results do not imply that the policy was unsuccessful in granting good 
students in poor schools a better chance to attend university, they do call attention to 
the potential for manipulation on the part of the high school when non-standardized 


measures of performance, such as GPA, are used in university admission criteria. 
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Appendix 


Figure A.1: 8th grade GPA by quintile (last grade level without university admis- 
sion consequences), cohorts 2011, 2012, 2013, and 2014. 
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Figure A.2: Yearly GPA by quintile, cohorts 2011, 2012, 2013, and 2014. 
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Table A.1: Filters applied to the data. 
Annual 
Sample Individuals observations 


Original sample: Students finishing high school 1.064.269 3.638.991 
between 2011 and 2014, one observation per level 


Sample after restricting to balanced panel 755.770 3.023.080 
(5 observations per individual) 


Sample after dropping any years in which a 
student did not pass* and individuals with 726.637 2.906.544 
contradictory level data 


*GPAs from repeated levels are excluded from university application GPAs in Chile. 
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Figure A.3: Yearly change in GPA by quintile, years 2008-2014, full sample. 
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(d) Between 11th and 12th grade 


Table A.2: Effect of ranking score introduction on yearly change in GPA, estima- 
tion with simple grade level dummies, cohorts 2011-2014. 


() 2) 
AGPA AGPA 

VARIABLES per grade __ per grade 
Ranking score in place 0.0466***  0.0467*** 
(0.00252) (0.00427) 
10th grade 0.280*** 0.280*** 
(0.00540) (0.00551) 
11th grade 0.243*** 0.243*** 
(0.00634) (0.00663) 
12th grade OBO "* 0:30774* 
(0.00609) (0.00682) 

Cohort 2012 0.00447*** 
(0.00126) 

Cohort 2013 0.00663*** 
(0.00215) 

Cohort 2014 0.0115*** 
(0.00320) 

Low SES student -0.0245*** 
(0.00145) 

Male student -0.00627*** 
(0.00170) 

Private school, subsidized -0.00860** 
(0.00369) 

Private school, non-subsidized -0.0407*** 
(0.00900) 

Constant -0.246*** = -0.234*** 


(0.00545) (0.00666) 


Observations 2,906,544 2,906,544 
R-squared 0.125 0.126 
Clustered standard errors at the school level in parentheses. 
*** D<0.1, ** p<0.05, *** p<0.01. 
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Table A.3: OLS and 2SLS effect of ranking score bonus including linear controls for 


8th grade GPA. 


@ © 8) 
AGPA Marginal AGPA 
VARIABLES OLS bonus 2SLS 
Marginal bonus -0.0389*** 3.87e-05 
(0.00100) (0.00287) 
Simulated marginal bonus 0.248*** 
(0.00543) 
Ranking score in place 0.0691*** e276" 0.0539*** 
(0.00410) (0.00462) (0.00413) 
Low SES student -0.0291***  0.0153*** = -0.0302*** 
(0.00122) (0.00170) (0.00123) 
Male student -0.0189***  0.00372*** -0.0191*** 
(0.00154) (0.00143) (0.00155) 
Private school, subsidized -0.00684**  -0.00606** -0.00660** 
(0.00312) (0.00279) (0.00312) 
Private school, non-subsidized -0.0369***  -0.0185***  -0.0363*** 
(0.00790) (0.00638) (0.00789) 
10th grade 0.278*** = -0.00737*** — 0.278*** 
(0.00485) (0.000719) (0.00486) 
11th grade 0.239*** -0.000734 0.239*** 
(0.00613) (0.00102) (0.00614) 
12th grade 0.303*** 0.0222*** 0.302*** 
(0.00595) (0.00184) (0.00597) 
(GPA 8th - Mean GPA 8t")-L9 —-0.292*** —0.00156*** — -0.292*** 
(0.00499) (0.000283) — (0.00499) 
(GPA 8b - Mean GPA 8") - L10 -0.0582*** — 0.0359*** — -0.0606*** 
(0.00153) (0.00211) (0.00156) 
(GPA 8tt - Mean GPA 8") - L11 -0.0638*** — 0.0892*** — -0.0695*** 
(0.00270) (0.00407) (0.00277) 
(GPA 85 - Mean GPA 8**).L12 -0.0159***  0.172*** — -0.0249*** 
(0.00219) (0.00587) (0.00230) 
Constant -0.221*** -0.00264 -0.221*** 
(0.00546) (0.00181) (0.00545) 
Observations 2,906,544 2,906,544 2,906,544 
R-squared 0.166 0.215 0.164 
Cohort FE Yes Yes Yes 


Clustered standard errors at the school level in parentheses. 
*** D<0.1, ** p<0.05, *** p<0.01. 
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